Multi-layer optimization for a multi-sided network service

ABSTRACT

A computing system generates recommendations for users within the context of a network service. To account for objectives of various users associated with the network service, some of which may not reach optimality at the same time, the computing system generates values associated with each of the objectives separately. For example, for each objective, the system may train a computer model to produce a representative value. To generate a recommendation of an entity for a user, the system uses the generated objective values as inputs to an optimization algorithm. The optimization step may use linear programming or quadratic programming to generate a recommendation score, for example. This two-step process allows the system to account for multiple objectives and makes the system easily adaptable to change when the set of objectives is updated.

BACKGROUND Field of Art

This disclosure relates generally to content recommendation, and inparticular to optimizing recommendations in view of a plurality ofobjectives in a dynamic marketplace.

Description of Art

Dynamic marketplaces are improved when all marketplace participantsbenefit from their participation in the marketplace. Additionally, asystem that operates a platform for such a marketplace may benefit whenthe marketplace participants behave in certain ways. For example, asystem may operate a food delivery marketplace by providing a platformthat connects restaurants, food delivery partners, and consumers.However, it can be difficult to simultaneously account for differentrequirements and optimal conditions for all of the marketplaceparticipants.

SUMMARY

A recommendation system for a marketplace platform manages optimizationof objectives for multiple categories of marketplace participants. Thesystem provides recommendations of content items to present toconsumers, where a consumer is a user of a network service such as amarketplace platform. The recommendations may be personalized for eachconsumer and they may be presented in a way that addresses the challengeof meeting multiple objectives in real time.

While an example of a food delivery service is presented herein as anexample marketplace in which the recommendation system may be used, inother examples, the recommendation system and associated marketplaceplatform may be used in a variety of different marketplace environments,and the marketplace environments may have more or fewer objectives.

A first layer of the system includes models for each objective. In oneembodiment, one of the models can be used to predict a probability thata consumer will order from a restaurant. This “demand prediction” modelmay accept data including user profile data about a consumer, data aboutthe restaurant, and contextual data as inputs. The demand predictionmodel can then determine a prediction score for a consumer-restaurantpairing, where the prediction score is indicative, for example, of thelikelihood that the consumer will choose to order food from therestaurant.

Other objective models in the first layer of the system may includemodels for generating various other predictive values. In oneembodiment, the objective models include models of market fairness,gross earnings for the platform, delivery partner reliability, andconsumer happiness. The objective models may be trained or unsupervisedcomputer models, neural networks, or specific algorithms developed forgenerating particular objective values. In one embodiment, certainobjective values may be predetermined, e.g., as set by a systemadministrator or manager of the marketplace platform system.

A second layer of the system generates recommendation scores (e.g., foran order in which the restaurants should be displayed to the consumer)by optimizing among the values output by the multiple objective modelsin the first layer. This second layer may thus be used to solve aconstrained optimization problem, for example using linear or quadraticprogramming. Thresholds may be set for some of the objective valueswhile the system optimizes for other objectives.

Modeling the various objective values independently from one another inthe first layer of the system gives the system flexibility to add orremove different objective models without interfering with existingobjectives, e.g., without a need to retrain existing objective models.For example, a system administrator could introduce a new objectivemodel to the system and would only need to adjust the optimizationalgorithm to account for the additional or changed inputs. Themulti-layer scheme also allows the various objective values to begenerated in real time, further enabling the final recommendation of thesystem to account for the most current data (e.g., restaurants that areopen now, available menus, available delivery partners, etc.).

The second layer of the system outputs a personalized recommendation ofa restaurant for a consumer. By generating a personalized recommendationscore for a consumer with respect to a set of restaurants, the systemcan determine a current ranking of restaurants to present to theparticular consumer. The ranking information may be used to determine inwhat format or in what order restaurants should be presented to theconsumer, for example, via an application on a client device like asmart phone, tablet, or computer that the consumer may be using toaccess the food delivery platform.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a high-level block diagram of a system environment andarchitecture of a system, in accordance with an example embodiment.

FIG. 2 is a data flow diagram illustrating a process for generating arecommendation, in accordance with an example embodiment.

FIG. 3 is a flowchart illustrating a method of generating and presentingrecommendations to users, in accordance with an example embodiment.

FIG. 4 is a block diagram illustrating components of an example machinefor reading and executing instructions from a machine-readable medium,in accordance with an example embodiment.

The figures depict an embodiment of the invention for purposes ofillustration only. One skilled in the art will readily recognize fromthe following description that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles of the invention described herein.

DETAILED DESCRIPTION

For a marketplace to work efficiently, a network service, such as a fooddelivery marketplace system, can be configured to consider the interestsof each category of marketplace participant. In some embodiments, thesystem may do this by providing a fair marketplace with exposure forentities associated with the marketplace. Restaurants are providedherein as an example of a type of entity that might be associated with afood delivery marketplace system. For example, the system providesexposure to restaurants so that the restaurants each have a chance toreceive enough orders, and so that the delivery partners all have enoughorders to deliver while also being available to deliver all consumerorders in a timely manner. The system will benefit from consumers whorepeatedly choose to order food via the food delivery marketplace, andwill thus benefit from measures that lead to consumer happiness andretention.

As the example of a three-sided food delivery marketplace demonstrates,providing a marketplace platform that optimizes metrics related to theobjectives of all of the categories of participants can be a challengingand complex undertaking. This is especially true as the dynamics of suchmarketplaces may change rapidly, e.g., as participants from differentcategories of participants become active on the platform at differenttimes throughout a day.

In particular, objectives associated with the different categories ofparticipants are frequently in some degree of competition. That is,optimizing one objective may not also optimize the others. For example,in the case of the food delivery marketplace, there may be a tradeoffbetween optimizing an objective of high consumer conversion rate andoptimizing an objective of marketplace fairness for the restaurants.Optimizing only for consumer conversion rate may result in a few popularrestaurants being repeatedly included at the top of a list ofrecommended restaurants by the marketplace platform overseen by thesystem since consumers are known to order from those restaurants.However, this reduces marketplace fairness for new restaurants that maynot yet be well known. Conversely, optimizing only for marketplacefairness by ensuring equal exposure of every restaurant may result in alower consumer conversion rate as consumers may be less likely to findrestaurants with food they want to order.

A computing system provides a marketplace platform for a marketplace(e.g., a network service), such as a dynamic marketplace. As describedherein, a dynamic marketplace may be a marketplace in which marketplaceparticipants (e.g., providers, sellers, consumers, browsers, deliverypartners, etc.) may enter/view or leave the marketplace independentlyand at different times. For example, a food delivery marketplace mayinclude three categories of marketplace participant: restaurants,consumers, and delivery partners. A consumer may enter the marketplace,such as by opening an application on a user device to access themarketplace platform, when the consumer wants to order food, and mayleave the marketplace after placing an order with a restaurant. Adelivery partner may enter the marketplace to signal availability formaking deliveries and may leave the marketplace when the deliverypartner is no longer available to make deliveries. The restaurants maybe in the marketplace during their hours of business. Dynamics ofmarketplace participants can be further complicated by other factors(e.g., distance between restaurant and consumer) and number and type ofcategories of participants may vary among different types ofmarketplaces.

The system generates recommendations (e.g., recommending restaurants toconsumers) while also accounting for variability in the nature,presence, and needs of marketplace participants. In particular, thesystem generates recommendations that can incorporate objectives formultiple categories of marketplace participants, such as consumerconversion rate for consumers, or marketplace fairness for restaurants.At a first stage, the system applies a plurality of objective models tofeature values related to the marketplace participants. The objectivemodels, operating independently of one another, produce objective valuesthat represent different objectives that the marketplace participantsmay have. At a second stage, the system uses linear programming,quadratic programming, or another strategy to generate a recommendationthat optimizes the various objective values. In some cases, the systemmay also account for threshold optimization values for differentobjectives, as it may not be possible for all objectives to besimultaneously optimal.

FIG. 1 is a high-level block diagram of a system environment andarchitecture of a system, in accordance with an example embodiment. FIG.1 includes a user device 100, a network 120, and the system 130. Forclarity, only one user device 100 is shown in FIG. 1. Alternateembodiments of the system environment can have any number of userdevices 100 as well as multiple systems 130. The functions performed bythe various entities of FIG. 1 may vary in different embodiments. Forexample, in one embodiment, the system 130 may coordinate food deliveryfrom a restaurant to a customer via a delivery partner. Multiple userdevices 100 operating in the system environment may be used for avariety of purposes by different users who are acting as participants ina marketplace for which the system 130 provides a platform. The system130 generates recommendations for consumers in the marketplace. In theexample of the food delivery marketplace, the system 130 may generatescores for restaurants that indicate whether the restaurant should berecommended to a particular consumer.

A user (e.g., a consumer, restaurant representative, or deliverypartner) may enter location information, request deliveries, orotherwise communicate with the system 130 through a user device 100.User devices 100 can be personal or mobile computing devices, such assmartphones, tablets, notebook computers, or desktop computers. In someembodiments, the user device 100 executes a client application that usesan application programming interface (API) to communicate with thesystem 130 through the network 120.

In some embodiments, a consumer can make a request to the system 130through operation of the user device 100. For example, a food deliveryrequest may include one or more of user identification information, afood order, and a current location or the food delivery location for aconsumer. The current location of a user device 100 and/or the fooddelivery location may be designated by the user, or detected using alocation sensor of the user device 100 (e.g., a global positioningsystem (GPS) receiver).

In one embodiment, a user device 100 may send user information (e.g.,consumer location, etc.) to the system 130 when an applicationassociated with the system 130 is open on the user device. In this way,the system 130 can generate recommendations that are appropriate for theuser's current situation. In some embodiments, the system 130 mayrequest user data user information when the application associated withthe system 130 is open on the user device 100 and communicating with thesystem 130. The user device 100 responds with all or a subset of therequested user information.

User devices 100 may interact with the system 130 through clientapplications configured to interact with the system 130. The clientapplications of the user devices 100 can present information receivedfrom the system 130 on a user interface, such as a list of restaurantsand menus, a map of the geographic region of a user, and the currentlocation of the user device 100. The client application running on theuser device 100 may be able to determine the current location of theuser device 100 and provide the current location to the system 130.

The user devices 100 can communicate with the system 130 via the network120 which may comprise any combination of local area and wide areanetworks employing wired of wireless communication links. In someembodiments, all or some of the communication on the network 120 may beencrypted.

As illustrated in FIG. 1, the system 130 includes various modules anddata stores for use in providing a marketplace platform and forgenerating restaurant recommendations for consumers. The system 130comprises a model generator 140, a model store 150, one or moreobjective modules 160, and an optimization module 170. Computercomponents such as web servers, network interfaces, security functions,load balancers, failover servers, management and network operationsconsoles, and the like are not shown so as to not obscure the details ofthe system architecture. Additionally, the system 130 may contain more,fewer, or different components than those shown in FIG. 1 and thefunctionality of the components as described herein may be distributeddifferently from the description herein.

The model generator 140 trains models for quantifying a variety ofobjectives. The computer models may be supervised or unsupervisedmodels. Supervised models may be trained by the model generator 140using labeled training data (e.g., data from past interactions of marketparticipants with the system 130). In some embodiments, the modelgenerator 140 manages the generation of different kinds of models fordifferent objectives, such as consumer conversion rate or marketplacefairness. In some embodiments the model generator 140 retrains orupdates the models for some or all of the objectives. Model updates mayoccur when new training data is obtained, periodically, or on anotherschedule (e.g., weekly, daily, etc.).

Models and algorithms that generate values associated with differentobjectives may be stored in the model store 150. The model store 150 isa data store that in various embodiments may store labeled trainingdata, unlabeled training data, and/or trained computer models (e.g.,decision trees, neural networks, etc.).

The objective modules 160 use models and algorithms from the model store150 to generate objective values, that is, values related to objectivesthat the different marketplace participants might have. In someembodiments, one objective module 160 may handle the application ofmodels for each of a plurality of objectives. In one embodiment, asshown herein for the sake of clarity, separate objective modules 160manage the generation of values for each of the objectives. For example,one objective module 160 may generate a score of market fairness using amarket fairness model from the model store 150 and another objectivemodule 160 may generate a value representative of predicted grossearnings using another model from the model store 150.

The optimization module 170 generates a recommendation score (e.g.,indicative of whether a restaurant should be recommended to a user, anorder in which to recommend a restaurant to a user, etc.) based on theone or more objective values generated by the objective modules 160. Theoptimization module 170 uses quadratic programming, linear programming,or another optimization technique to determine a recommendation scorethat optimizes among the objectives. In some embodiments, theoptimization module is preprogrammed with quadratic programming rules(e.g., equations) that define how to make trade-offs when the variousobjectives are not all optimized by the same input. The optimizationmodule 170 may include threshold optimization values for each of theobjectives represented by objective modules 160. For example, theoptimization module 170 may include a rule to maximize a value of one ofthe objectives (e.g., restaurant gross bookings) while settingconstraints on the amount of variation that is acceptable for the valuesof each of the other objectives (e.g., customer happiness). In oneembodiment, a resulting optimization problem yields an overallrecommendation score as a combination of the values associated with theobjectives, each value having a weight associated with a thresholdoptimization value that represents the amount of acceptable variation.The threshold optimization values, as well as other equations andinequalities related to the quadratic programming may be preprogrammed(e.g., by a system administrator or developer of the system 130).

The modular configuration of the system 130 makes it possible forobjectives to be added to or removed as inputs to generatingrecommendation scores without having to retrain a model each time.Instead, to incorporate a new objective, the model generator 140 trainsa model to produce values related to the new objective and the system130 may allocate an objective module 160 to manage the use of the newmodel. Additionally, the optimization module 170 may be updated toincorporate a new input value (as received from the objective module 160relating to the new objective) into the quadratic or linear programmingscheme. For example, equations used by the optimization module 170 maybe updated to include a variable representative of the new input valueand threshold values related to the new objective may also be added tothe optimization scheme.

FIG. 2 is a data flow diagram illustrating a process for generating arecommendation, in accordance with an example embodiment. In particular,the process of FIG. 2 generates a recommendation 220 in the form of ascore indicating, for a given consumer-restaurant pair, whether therestaurant should be recommended to the particular consumer. Forexample, in one embodiment, after generating recommendation scores for auser with respect to multiple restaurants, a ranking of therecommendation scores determines an order in which a client applicationshould present the restaurants to the consumer via the consumer's userdevice 100.

Features 210 are input into the objective modules 160. Differentobjective modules may require different features 210. FIG.2 depicts ademand prediction module 160A, that is, an objective module 160 thatgenerates a value for the objective of optimizing consumer demand. Thedemand prediction module 160A generates a likelihood score thatrepresents a predicted likelihood that a consumer will find an entity(e.g., a restaurant) to be favorable, which may lead, for example, tothe consumer ordering from a restaurant. The demand prediction module160A accepts user features 210A for a consumer, restaurant features 210Bfor a restaurant, and contextual features 210C as inputs.

Objective modules 160 may use a variety of features 210 that areobtained by the system 130. User features 210A may include informationabout a user device 100 or about a user of the user device (e.g., aconsumer, delivery partner, or the like). Particular user features 210Arequired by an objective module 160 may vary based on the nature of theobjective. Examples of user features 210A as may be used by the demandprediction module 160A include: past interactions of the consumer withthe marketplace, past interactions of the consumer with the restaurant,restaurant links the consumer has clicked on in the past, restaurantsthe consumer has ordered from in the past, dishes the consumer hasordered in the past, dishes that similar consumers (e.g., like-minded,located nearby, etc.) order, and so forth.

In the example case of a food delivery marketplace, the demandprediction module 160A also receives restaurant features 210B as inputs.Examples of restaurant features 210B include: dishes on a menu of therestaurant, most popular dishes, a number of views of the restaurant, anumber of impressions the restaurant has received, a number of ordersthe restaurant has received, an estimated time that food will be ready,a type of cuisine, hours of operation, and restaurant location.

As an additional example of inputs, the demand prediction module 160Aaccepts contextual features 210C. Contextual features are features thatprovide information about the context of a potential food deliverytransaction. For example, the type of food a consumer is likely to ordermay vary based on the time of day that the consumer accesses themarketplace via a client application on a user device 100 (e.g., at 9a.m. a consumer may want to order breakfast food). Examples ofcontextual features 210C for the demand prediction module 160A include:a current location of the consumer, a delivery address, the time of day,the day of the week, whether it is a holiday, the meal period (e.g.,breakfast, lunch, dinner), the weather at the user's location, and thelike.

In addition to the demand prediction module 160A, the example of FIG. 2shows four other objective modules 160. The four additional modulesdepicted for use in the recommendation process include a market fairnessmodule 160B, a gross earnings module 160C, a partner reliability module160D, and a customer happiness module 160E. Objective modules 160 mayeach require differing sets of input data, as they are producing resultsrelated to different objectives.

The market fairness module 160B may generate one or more values relatedto how fair the marketplace is, that is, related to how even thedistribution of custom is among a set of restaurants. In one embodiment,the market fairness module may produce a fairness score that indicateshow presentation of the restaurant to the consumer will affectmarketplace fairness for other restaurants. For example, the marketplacefairness module 160B may accept among its inputs a set of restaurants inthe marketplace that are local to a consumer (e.g., within deliverydistance), and may output a score for the restaurant in question thatindicates how frequently the restaurant receives exposure as comparedwith the other restaurants in the set. In another example embodiment,the marketplace fairness module may output a list of the restaurantsordered by frequency of delivery orders received. Such information couldbe used by the system 130 to promote restaurants that do not currentlyreceive many delivery orders, for example by displaying informationabout new and relatively unknown restaurants more prominently than(e.g., earlier in a set of search results than) information aboutestablished popular restaurants that already receive many orders and alarge amount of exposure.

The gross earnings module 160C may generate values that can be used tooptimize for increased income, e.g., for the organization sponsoring thesystem 130 and for the restaurants. The objective related to the grossearnings module 160C might be to maximize earnings for the system or forthe restaurants. In one embodiment, the gross earnings module 160C mayreceive information about the cost of different items on a restaurantmenu as input, and may output one or more values related to an amount ofmoney the system 130 and the restaurant is likely to make throughdeliveries. For example, given a particular restaurant-consumer pairing,the gross earnings module 160C may output a prediction of the amount ofmoney the consumer is likely to spend if the consumer was to order fromthe particular restaurant.

The partner reliability module 160D may generate values or scoresrelated to one or more delivery partners. For example, the partnerreliability module 160D may be associated with the objective of makingsure that each delivery partner in a marketplace is assigned enoughdeliveries and doesn't sit idle when available for making deliveries. Inone embodiment, the partner reliability module 160D may acceptinformation about the location of delivery partners, the availability ofthe delivery partners in the marketplace, how often the delivery partnerdelivers ahead or behind schedule, restaurants the delivery partnertends to deliver from, and so forth. In one embodiment, the partnerreliability module 160D may output a partner reliability score thatindicates an extent to which a delivery from a given restaurant to agiven consumer will contribute to evening out the delivery load amongall available delivery partners in an area. For example, a restaurantthat is currently nearest to a delivery partner who has not beenrecently assigned a delivery may receive a higher partner reliabilityscore than a restaurant that is near delivery partners who have beenbusy making many deliveries. In another embodiment, the partnerreliability module 160D may output a list of available delivery partnersin the area ordered by how recently each delivery partner has made adelivery.

The customer happiness module 160E is related to the objective ofkeeping consumers happy with the food and services they receive so thatthey return to the system 130 in the future for their food deliveryneeds. Accordingly, the customer happiness module 160E may receiveinformation about restaurant ratings, and whether consumers make repeatorders from restaurants, as input. In one example embodiment, thecustomer happiness module 160E may output a likelihood score related towhether a consumer is likely to have a positive experience ordering froma given restaurant and whether the consumer is likely to order fooddelivery from the system 130 again if the consumer orders from the givenrestaurant.

The optimization module 170 generates a recommendation 220 score for theconsumer-restaurant pair. That is, the optimization module 170 outputs apersonalized recommendation score of a restaurant for a consumer. Therecommendation score accounts for the multiple objectives as representedby the values produced at the objective modules 160. To incorporateinformation from all of the objectives, the optimization module 170applies the values output by the various objective modules 160 to amodel or algorithm (e.g., linear programming or quadratic programmingalgorithm) that can generate the recommendation 220. The algorithm maybe preprogrammed by an administrator or developer of the system 130, forexample, when an objective and accompanying objective module 160 isadded or removed from the system 130. Adjusting the algorithm of theoptimization module 170 to a different set of objective modules 160 mayinclude introducing new threshold optimization values associated withthe different objectives. In one example embodiment, an algorithm forthe optimization module 170 generates a sum of the values produced bythe market fairness module 160B, the gross earnings module 160C, thepartner reliability module 160D, and the customer happiness module 160E,each objective value in the sum being multiplied by a weight associatedwith a threshold optimization value, and multiples the result by theoutput value of the demand prediction module 160A to generate arecommendation 220 value.

FIG. 3 is a flowchart illustrating a method of generating and presentingrecommendations to users, in accordance with an example embodiment. Thesystem 130 obtains 310 features related to a consumer, features relatedto a set of restaurants, and contextual features. In some embodiments,the features may be “real-time” values in that they may be the mostrecently collected values, ensuring that the recommendation 220 will bemade in view of present conditions (e.g., food for the appropriate mealperiod, and restaurants that are currently open, may be recommended).For each of the restaurants in the set of restaurants, the system 130generates 320 a likelihood score that represents a likelihood that theconsumer will order from the restaurant.

The system 130 may additionally obtain 330 features related to a set ofmarketplace objectives (e.g., user features, restaurant features, orcontextual features that may be used as inputs to objective module 160).As with the features related to the consumer and the set of restaurants,the features that are related to marketplace objectives may be real-timevalues. Using the features related to the marketplace objectives, thesystem 130 generates 340 a set of objective values. For example, eachobjective module 160 may apply a trained computer model to a set ofinput data to generate a value related to the associated objective.

The system 130 generates 350 a recommendation score for each of therestaurants in the set of restaurants, the recommendation score being avalue for a restaurant with respect to the particular consumer. Therecommendation score is based on the generated likelihood score and onone or more of the values in the set of generated objective values. Therecommendation scores for each of the restaurants may be used to presentinformation about the restaurants to the consumer. The system 130displays 360 information about one or more of the restaurants from theset of restaurants at a user device 100 of the consumer. For example,the order in which information about restaurants is displayed on ascreen of the user device 100 may be based on the recommendation scoresassociated with each of the restaurants.

FIG. 4 is a block diagram illustrating components of an example machineable to read instructions from a machine-readable medium and executethem in one or more processors (or controllers). Specifically, FIG. 4shows a diagrammatic representation of a system 130 in the example formof a computer system 400. The computer system 400 can be used to executeinstructions 424 (e.g., program code or software) for causing themachine to perform any one or more of the methodologies (or processes)described herein. In alternative embodiments, the machine operates as astandalone device or a connected (e.g., networked) device that connectsto other machines. In a networked deployment, the machine may operate inthe capacity of a server machine or a client machine in a server-clientnetwork environment, or as a peer machine in a peer-to-peer (ordistributed) network environment.

The machine may be a server computer, a client computer, a personalcomputer (PC), a tablet PC, a set-top box (STB), a smartphone, aninternet of things (IoT) appliance, a network router, switch or bridge,or any machine capable of executing instructions 424 (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute instructions 424 to perform any one or more of themethodologies discussed herein.

The example computer system 400 includes one or more processing units(generally processor 402). The processor 402 is, for example, a centralprocessing unit (CPU), a graphics processing unit (GPU), a digitalsignal processor (DSP), a controller, a state machine, one or moreapplication specific integrated circuits (ASICs), one or moreradio-frequency integrated circuits (RFICs), or any combination ofthese. The computer system 400 also includes a main memory 404. Thecomputer system may include a storage unit 416. The processor 402,memory 404, and the storage unit 416 communicate via a bus 408.

In addition, the computer system 406 can include a static memory 406, agraphics display 410 (e.g., to drive a plasma display panel (PDP), aliquid crystal display (LCD), or a projector). The computer system 400may also include alphanumeric input device 412 (e.g., a keyboard), acursor control device 414 (e.g., a mouse, a trackball, a joystick, amotion sensor, or other pointing instrument), a signal generation device418 (e.g., a speaker), and a network interface device 420, which alsoare configured to communicate via the bus 408.

The storage unit 416 includes a machine-readable medium 422 on which isstored instructions 424 (e.g., software) embodying any one or more ofthe methodologies or functions described herein. For example, theinstructions 424 may include instructions for implementing thefunctionalities of the model generator 140, the optimization module 170,or of the one or more objective module 160. The instructions 424 mayalso reside, completely or at least partially, within the main memory404 or within the processor 402 (e.g., within a processor's cachememory) during execution thereof by the computer system 400, the mainmemory 404 and the processor 402 also constituting machine-readablemedia. The instructions 424 may be transmitted or received over anetwork 426 via the network interface device 420.

While machine-readable medium 422 is shown in an example embodiment tobe a single medium, the term “machine-readable medium” should be takento include a single medium or multiple media (e.g., a centralized ordistributed database, or associated caches and servers) able to storethe instructions 424. The term “machine-readable medium” shall also betaken to include any medium that is capable of storing instructions 424for execution by the machine and that cause the machine to perform anyone or more of the methodologies disclosed herein. The term“machine-readable medium” includes, but not be limited to, datarepositories in the form of solid-state memories, optical media, andmagnetic media.

The foregoing description of the embodiments has been presented for thepurpose of illustration; it is not intended to be exhaustive or to limitthe patent rights to the precise forms disclosed. Persons skilled in therelevant art can appreciate that many modifications and variations arepossible in light of the above disclosure.

Some portions of this description describe the embodiments in terms ofalgorithms and symbolic representations of operations on information.These algorithmic descriptions and representations are commonly used bythose skilled in the data processing arts to convey the substance oftheir work effectively to others skilled in the art. These operations,while described functionally, computationally, or logically, areunderstood to be implemented by computer programs or equivalentelectrical circuits, microcode, or the like. Furthermore, it has alsoproven convenient at times, to refer to these arrangements of operationsas modules, without loss of generality. The described operations andtheir associated modules may be embodied in software, firmware,hardware, or any combinations thereof.

Any of the steps, operations, or processes described herein may beperformed or implemented with one or more hardware or software modules,alone or in combination with other devices. In one embodiment, asoftware module is implemented with a computer program productcomprising a computer-readable medium containing computer program code,which can be executed by one or more computer processors for performingany or all of the steps, operations, or processes described.

Embodiments may also relate to an apparatus for performing theoperations herein. This apparatus may be specially constructed for therequired purposes, and/or it may comprise a general-purpose computingdevice selectively activated or reconfigured by a computer programstored in the computer. Such a computer program may be stored in anon-transitory, tangible computer readable storage medium, or any typeof media suitable for storing electronic instructions, which may becoupled to a computer system bus. Furthermore, any computing systemsreferred to in the specification may include a single processor or maybe architectures employing multiple processor designs for increasedcomputing capability.

Embodiments may also relate to a product that is produced by a computingprocess described herein. Such a product may comprise informationresulting from a computing process, where the information is stored on anon-transitory, tangible computer readable storage medium and mayinclude any embodiment of a computer program product or other datacombination described herein.

Finally, the language used in the specification has been principallyselected for readability and instructional purposes, and it may not havebeen selected to delineate or circumscribe the inventive subject matter.It is therefore intended that the scope of the patent rights be limitednot by this detailed description, but rather by any claims that issue onan application based hereon. Accordingly, the disclosure of theembodiments is intended to be illustrative, but not limiting, of thescope of the patent rights, which is set forth in the following claims.

What is claimed is:
 1. A non-transitory computer-readable storage mediumstoring computer program instructions executable by one or moreprocessors of a computing system to perform steps comprising: for eachof a set of multiple different objectives for the network service:obtaining a threshold optimization value for the objective; accessinglabeled training data associated with the objective; training a machinelearning model, using the labeled training data, to produce an objectivevalue related to the objective; adjusting an optimization module togenerate a user recommendation score using the objective values outputby the trained machine learning models associated with the set ofobjectives, the recommendation score based on an optimization of each ofthe objective values toward the obtained threshold optimization valuefor the objective; obtaining current features related to a user of anetwork service and a set of entities associated with the networkservice; obtaining current features related to the set of objectives forthe network service; generating, by providing the obtained currentfeatures related to the set of objectives to the trained machinelearning models, a set of objective values, each of the set of objectivevalues associated with an objective of the set of objectives, wherein atleast one of the objective values is a marketplace fairness score thatrepresents how frequently an entity receives exposure to client devicesof users compared with other entities associated with the networkservice; for each entity of the set of entities: generating a likelihoodscore quantifying whether the user will find the entity to be favorable,the likelihood score based on the current features related to the userand the current features related to the entity; generating a userrecommendation score for the entity, the user recommendation score basedon the likelihood score and the set of objective values; and displayinginformation about one or more of the entities to a client device of theuser.
 2. The non-transitory computer-readable storage medium of claim 1,the steps further comprising: training a first model associated with oneof the multiple different objectives to generate a probability that auser will find an entity to be favorable, the training taking as input:information about the entity, information about the user, and contextualinformation.
 3. The non-transitory computer-readable storage medium ofclaim 1, wherein obtaining current features related to a user of anetwork service comprises: responsive to a user opening an applicationon a client device: requesting contextual user data from the clientdevice; and receiving a subset of the requested contextual user datafrom the client device.
 4. The non-transitory computer-readable storagemedium of claim 1, wherein the format of the displayed information isdetermined using the recommendation scores of the entities.
 5. Thenon-transitory computer-readable storage medium of claim 1, whereingenerating the user recommendation score comprises applying, by theoptimization module, a constrained quadratic programming algorithm tothe generated objective values and the likelihood score.
 6. Thenon-transitory computer-readable storage medium of claim 5, whereinconstraints applied to the quadratic programming algorithm include thethreshold optimization values associated with each of the objectives forthe network service.
 7. A computer-implemented method, performed by oneor more computer processors, for recommending entities to users on aplatform for a network service, the computer-implemented methodcomprising: for each of a set of multiple different objectives for thenetwork service: obtaining a threshold optimization value for theobjective; accessing labeled training data associated with theobjective; training a machine learning model, using the labeled trainingdata, to produce an objective value related to the objective; adjustingan optimization module to generate a user recommendation score using theobjective values output by the trained machine learning modelsassociated with the set of objectives, the recommendation score based onan optimization of each of the objective values toward the obtainedthreshold optimization value for the objective; obtaining currentfeatures related to a user of the network service and a set of entitiesassociated with the network service; obtaining current features relatedto the set of objectives for the network service; generating, byproviding the obtained current features related to the set of objectivesto the trained machine learning models, a set of objective values, eachof the set of objective values associated with an objective of the setof objectives, wherein at least one of the objective values is amarketplace fairness score that represents how frequently an entityreceives exposure to client devices of users compared with otherentities associated with the network service; for each entity of the setof entities: generating a likelihood score quantifying whether the userwill find the entity to be favorable, the likelihood score based on thecurrent features related to the user and the current features related tothe entity; generating a user recommendation score for the entity, theuser recommendation score based on the likelihood score and the set ofobjective values; and displaying information about one or more of theentities to a client device of the user.
 8. The computer-implementedmethod of claim 7, further comprising: training a first model associatedwith one of the multiple different objectives to generate a probabilitythat a user will find an entity to be favorable, the training taking asinput: information about the entity, information about the user, andcontextual information.
 9. The computer-implemented method of claim 7,wherein obtaining current features related to a user of a networkservice comprises: responsive to a user opening an application on aclient device: requesting contextual user data from the client device;and receiving a subset of the requested contextual user data from theclient device.
 10. The computer-implemented method of claim 7, whereinthe format of the displayed information is determined using therecommendation scores of the entities.
 11. The computer-implementedmethod of claim 7, wherein generating a user recommendation score for anentity comprises applying, by the optimization module, a constrainedquadratic programming algorithm to the generated objective valuesrelated to the set of objectives and the likelihood score.
 12. Thecomputer-implemented method of claim 7, wherein the set of objectivesincludes: market fairness, gross earnings, delivery partner reliability,and consumer happiness.